1. Technical Field
The invention is related to an image-based rendering system and process for rendering novel views of a 3D scene, and more particularly, to a system and process for rendering these novel views using concentric mosaics.
2. Background Art
Traditional rendering approaches often involve constructing or recovering a complete geometric and photometric model of a 3D scene. These approaches are attractive as they allow graphics elements (such as new lights, shadows, etc.) to be readily added to the model. However, these approaches are also complex and computationally intensive. In many cases, it is desired to render views of a 3D scene more simply without the complexity and excessive processing requirements of a geometric/photometric rendering approach. If this is the case, a different method of rendering a view of the scene could be taken. Specifically, an image-based rendering process could be employed. One such image-based rendering approach foregoes the need for a geometric/photometric model of a scene, and instead uses Adelson and Bergen""s plenoptic function [AB91] to describe the scene. The original Adelson and Bergen work defined a 7D plenoptic function as the intensity of light rays passing through the camera center at every location (Vx, Vy, Vz) at every possible angle (xcex8, xcfx86), for every wavelength xcex, at every time t, i.e.,
P7=P(Vx,Vy,Vz,xcex8,xcfx86,xcex,t)xe2x80x83xe2x80x83(1)
It is also possible to add light source directions in the plenoptic function. [WHON97].
In recent years a number of image-based rendering techniques have been proposed to model and render real or synthetic scenes and objects based on simplifying the plenoptic function. For example, techniques have been proposed to construct a complete 5D plenoptic function by McMillan and Bishop [MB95]:
P5=P(Vx,Vy,Vz,xcex8,xcfx86)xe2x80x83xe2x80x83(2)
In this work two of the variables in the original equation are dropped, namely time t (therefore assuming a static environment) and light wavelength xcex (hence assuming a fixed lighting condition). In addition, this work introduced the concept of interpolating plenoptic functions from a few discrete samples using cylindrical panoramas.
A 4D parameterization of the plenoptic function has also been proposed. It was observed in both the Lightfield [LH96] and Lumigraph [GGSC96] systems that by staying outside a convex hull or bounding box of an object, the 5D complete plenoptic function can be simplified to a 4D lightfield plenoptic function, i.e.,
P4=P(u,v,s,z)xe2x80x83xe2x80x83(3)
where (u, v) and (s, z) parameterize two bounding planes of the convex hull. The reverse is also true if camera views are restricted inside a convex hull.
There has even been 2D simplifications proposed (e.g., cylindrical panoramas [Chen95] and spherical panoramas [SS97]) where the viewpoint in the scene is fixed and only viewing directions can be changed, i.e.,
P2=P(xcex8,xcfx86)xe2x80x83xe2x80x83(4)
The 2D embodiment of the plenoptic function is the easiest to construct. However, the 2D parameterization of the plenoptic function does not allow novel views from different viewpoints within the scene to be rendered. It would be possible to render novel views using the 5D or 4D embodiments of the plenoptic function. It is, however, very difficult to construct a 5D complete plenoptic function [MB95, KS96] because the feature correspondence is a difficult problem. In addition, while the 4D embodiment is not too difficult to construct because of its unique parameterization, the resulting lightfield is large even for a small object (therefore small convex hull). This presents problems when trying to render novel views, and to date a real-time walk-through a real 3D scene has not been fully demonstrated.
It is noted that in the preceding paragraphs, as well as in the remainder of this specification, the description refers to various individual publications identified by an alphanumeric designator contained within a pair of brackets. For example, such a reference may be identified by reciting, xe2x80x9creference [AB91]xe2x80x9d or simply xe2x80x9c[AB91]xe2x80x9d. Multiple references will be identified by a pair of brackets containing more than one designator, for example, [PH97, W+97, RG98, RPIA98, GH97]. A listing of the publications corresponding to each designator can be found at the end of the Detailed Description section.
The invention is directed toward an image-based rendering system and process for rendering novel views of a real or synthesized 3D scene based on a series of concentric mosaics depicting the scene.
In one embodiment of the present invention, each concentric mosaic represents a collection of consecutive slit images of the surrounding 3D scene taken in a direction tangent to a viewpoint on a circle on a plane within the scene. The mosaics are concentric in that the aforementioned circles on the plane are concentric. This system and process is based on a unique 3D parameterization of the previously described plenoptic function that will hereafter be referred to as concentric mosaics. Essentially, this parameterization requires that the concentric mosaics be created by constraining xe2x80x9ccameraxe2x80x9d motion on concentric circles in the circle plane within the 3D scene. Once created, the concentric mosaics can be used to create novel views of the 3D scene without explicitly recovering the scene geometry.
Specifically, each concentric mosaic is generated by capturing slit-shaped views of the 3D scene from different viewpoints along one of the concentric circles. Each captured slit image is uniquely indexed by its radial location (i.e., which concentric mosaic it belongs to) and its angle of rotation (i.e., where on the mosaic). The radial location is the radius of the concentric center on which the slit image was captured, and the angle of rotation is defined as the number of degrees from a prescribed beginning on the concentric circle to the viewpoint from which the slit image was captured. The height of the slit image reflects the vertical field of view of the capturing camera. Preferably, slit images are captured at each viewpoint on the concentric circle in both directions tangent to the circle. If so, the unique identification of each slit image also includes an indication as to which of the two tangential directions the slit image was captured.
Novel views from viewpoints on the aforementioned circle plane are rendered using the concentric mosaics composed of slit images. Specifically, the viewpoint of a novel view may be anywhere within a circular region defined by the intersection of the outermost concentric mosaic with the circle plane. The novel view image consists of a collection of side-by-side slit images, each of which is identified by a ray emanating from the viewpoint of the novel view on the circle plane and extending toward a unique one of the slit images needed to construct the novel view. Each of the rays associated with the slit images needed to construct the novel view will either coincide with a ray identifying one of the previously captured slit images in the concentric mosaics, or it will pass between two of the concentric circles on the circle plane. If it coincides, then the previously captured slit image associated with the coinciding ray can be used directly to construct part of the novel view. However, if the ray passes between two of the concentric circles of the plane, a new slit image is formed by interpolating between the two previously captured slit images associated with the rays originating from the adjacent concentric circles that are parallel to the non-coinciding ray of the novel view. This interpolation process preferably includes determining the distance separating the non-coinciding ray from each of the two adjacent parallel rays. Once these distances are determined, a ratio is computed for each slit image associated with the parallel rays based on the relative distance between the non-coinciding ray and each of the parallel rays. This ratio represents how much weight the slit image will be given in a blending process that combines the two slit images associated with the parallel rays to form an interpolated slit image. Once all the slit images needed to create the novel view have been identified, or created by interpolation, the novel view can be rendered by arranging the slit images in a side-by-side fashion in an order dictated by the rays associated with the novel view.
The rendering system and process can also be modified to mitigate the effects of potential distortions in the rendered images. These distortions can arise when objects depicted in a novel view are relatively close because the slit images used to construct the novel views were captured at viewpoints on the circle plane that are different from the viewpoint of the novel view. The potential distortion can be compensated for using a scaling process. This scaling process includes first determining the average distance from the viewpoint of each captured slit image to the objects of the 3D scene depicted therein. Then, for each captured slit image found to coincide with one of the slit images needed to construct a novel view, and for each interpolated slit image, the average distance from the objects of the 3D scene depicted in the captured slit image to the viewpoint of the novel view is determined. A ratio is computed by dividing the first distance by the second distance. This ratio is used to scale the vertical dimension of the slit image.
The system used to create the concentric mosaics typically uses multiple slit-image cameras that are rotated around a center point. In another embodiment of the present invention, a single camera, such as a video camera, may be used instead. At each rotation angle, a regular image is captured using the video camera, rather than a slit image, to obtain a sequence of images. A regular image includes multiple image lines wherein each image line is associated with a ray direction (a ray direction is an angle at which a light ray enters the lens of the camera). A concentric mosaic is formed by combining each image line having the same ray direction in the sequence of images. For example, the 20th image line taken from each image may be used to form a concentric mosaic. An image from a novel viewpoint may then be rendered using the same techniques described for the slit-image embodiment.
Using the single camera embodiment, an image from a novel viewpoint may be rendered without rebinning the images into concentric mosaics. Specifically, multiple ray directions are calculated from the novel viewpoint. For each ray direction, an image line is obtained from the sequence of images that has a ray direction substantially aligning with the ray direction from the novel viewpoint. The obtained image lines are then combined to render an image from the novel viewpoint.